Generalized Gaussian signal detection theory: A unified signal detection framework for confidence data analysis.

Human decision behavior entails a graded awareness of its certainty, known as a feeling of confidence. Until now, considerable interest has been paid to behavioral and computational dissociations of decision and confidence, which has raised an urgent need for measurement frameworks that can quantify the efficiency of confidence rating relative to decision accuracy (metacognitive efficiency). As a unique addition to such frameworks, we have developed a new signal detection theory paradigm utilizing the generalized Gaussian distribution (GGSDT). This framework evaluates the observer's metacognitive efficiency and internal standard deviation ratio through shape and scale parameters, respectively. The shape parameter quantifies the kurtosis of internal distributions and can practically be understood in reference to the proportion of the Gaussian ideal observer's confidence being disrupted with random guessing (metacognitive lapse rate). This interpretation holds largely irrespective of the contaminating effects of decision accuracy or operating characteristic asymmetry. Thus, the GGSDT enables hitherto unexplored research protocols (e.g., direct comparison of yes/no vs. forced-choice metacognitive efficiency), expected to find applications in various fields of behavioral science. This article provides a detailed walkthrough of the GGSDT analysis with an accompanying R package (ggsdt). (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Psychophysical measurement of perceived motion flow of naturalistic scenes.

The neural and computational mechanisms underlying visual motion perception have been extensively investigated over several decades, but little attempt has been made to measure and analyze, how human observers perceive the map of motion vectors, or optical flow, in complex naturalistic scenes. Here, we developed a psychophysical method to assess human-perceived motion flows using local vector matching and a flash probe. The estimated perceived flow for naturalistic movies agreed with the physically correct flow (ground truth) at many points, but also showed consistent deviations from the ground truth (flow illusions) at other points. Comparisons with the predictions of various computational models, including cutting-edge computer vision algorithms and coordinate transformation models, indicated that some flow illusions are attributable to lower-level factors such as spatiotemporal pooling and signal loss, while others reflect higher-level computations, including vector decomposition. Our study demonstrates a promising data-driven psychophysical paradigm for an advanced understanding of visual motion perception.

Coarse-to-fine interaction on perceived depth in compound grating.

To encode binocular disparity, the visual system uses a pair of left eye and right eye bandpass filters with either a position or a phase offset between them. Such pairs are considered to exit at multiple scales to encode a wide range of disparity. However, local disparity measurements by bandpass mechanisms can be ambiguous, particularly when the actual disparity is larger than a half-cycle of the preferred spatial frequency of the filter, which often occurs in fine scales. In this study, we investigated whether the visual system uses a coarse-to-fine interaction to resolve this ambiguity at finer scales for depth estimation from disparity. The stimuli were stereo grating patches composed of a target and comparison patterns. The target patterns contained spatial frequencies of 1 and 4 cycles per degree (cpd). The phase disparity of the low-frequency component was 0° (at the horopter), -90° (uncrossed), or 90° (crossed), and that of the high-frequency components was changed independent of the low-frequency disparity, in the range between -90° (uncrossed) and 90° (crossed). The observers' task was to indicate whether the target appeared closer to the comparison pattern, which always shared the disparity with the low-frequency component of the target. Regardless of whether the comparison pattern was a 1-cpd + 4-cpd compound or a 1-cpd simple grating, the perceived depth order of the target and the comparison varied in accordance with the phase disparity of the high-frequency component of the target. This effect occurred not only when the low-frequency component was at the horopter, but also when it contained a large disparity corresponding to one cycle of the high-frequency component (±90°). Our findings suggest a coarse-to-fine interaction in multiscale disparity processing, in which the depth interpretation of the high-frequency changes based on the disparity of the low-frequency component.

Does training with blurred images bring convolutional neural networks closer to humans with respect to robust object recognition and internal representations?

It has been suggested that perceiving blurry images in addition to sharp images contributes to the development of robust human visual processing. To computationally investigate the effect of exposure to blurry images, we trained convolutional neural networks (CNNs) on ImageNet object recognition with a variety of combinations of sharp and blurred images. In agreement with recent reports, mixed training on blurred and sharp images (B+S training) brings CNNs closer to humans with respect to robust object recognition against a change in image blur. B+S training also slightly reduces the texture bias of CNNs in recognition of shape-texture cue conflict images, but the effect is not strong enough to achieve human-level shape bias. Other tests also suggest that B+S training cannot produce robust human-like object recognition based on global configuration features. Using representational similarity analysis and zero-shot transfer learning, we also show that B+S-Net does not facilitate blur-robust object recognition through separate specialized sub-networks, one network for sharp images and another for blurry images, but through a single network analyzing image features common across sharp and blurry images. However, blur training alone does not automatically create a mechanism like the human brain in which sub-band information is integrated into a common representation. Our analysis suggests that experience with blurred images may help the human brain recognize objects in blurred images, but that alone does not lead to robust, human-like object recognition.

On the assumptions behind metacognitive measurements: Implications for theory and practice.

Theories of visual confidence have largely been grounded in the gaussian signal detection framework. This framework is so dominant that idiosyncratic consequences from this distributional assumption have remained unappreciated. This article reports systematic comparisons of the gaussian signal detection framework to its logistic counterpart in the measurement of metacognitive accuracy. Because of the difference in their distribution kurtosis, these frameworks are found to provide different perspectives regarding the efficiency of confidence rating relative to objective decision (the logistic model intrinsically gives greater meta-d'/d' ratio than the gaussian model). These frameworks can also provide opposing conclusions regarding the metacognitive inefficiency along the internal evidence continuum (whether meta-d' is larger or smaller for higher levels of confidence). Previous theories developed on these lines of analysis may need to be revisited as the gaussian and logistic metacognitive models received somewhat equivalent support in our quantitative model comparisons. Despite these discrepancies, however, we found that across-condition or across-participant comparisons of metacognitive measures are relatively robust against the distributional assumptions, which provides much assurance to conventional research practice. We hope this article promotes the awareness for the significance of hidden modeling assumptions, contributing to the cumulative development of the relevant field.

Visual discrimination of optical material properties: A large-scale study.

Complex visual processing involved in perceiving the object materials can be better elucidated by taking a variety of research approaches. Sharing stimulus and response data is an effective strategy to make the results of different studies directly comparable and can assist researchers with different backgrounds to jump into the field. Here, we constructed a database containing several sets of material images annotated with visual discrimination performance. We created the material images using physically based computer graphics techniques and conducted psychophysical experiments with them in both laboratory and crowdsourcing settings. The observer's task was to discriminate materials on one of six dimensions (gloss contrast, gloss distinctness of image, translucent vs. opaque, metal vs. plastic, metal vs. glass, and glossy vs. painted). The illumination consistency and object geometry were also varied. We used a nonverbal procedure (an oddity task) applicable for diverse use cases, such as cross-cultural, cross-species, clinical, or developmental studies. Results showed that the material discrimination depended on the illuminations and geometries and that the ability to discriminate the spatial consistency of specular highlights in glossiness perception showed larger individual differences than in other tasks. In addition, analysis of visual features showed that the parameters of higher order color texture statistics can partially, but not completely, explain task performance. The results obtained through crowdsourcing were highly correlated with those obtained in the laboratory, suggesting that our database can be used even when the experimental conditions are not strictly controlled in the laboratory. Several projects using our dataset are underway.

Motion Direction Discrimination with Tactile Random-Dot Kinematograms.

Motion detection is a fundamental sensory function for multiple modalities, including touch, but the mechanisms underlying tactile motion detection are not well understood. While previous findings supported the existence of high-level feature tracking, it remains unclear whether there also exist low-level motion sensing that directly detects a local spatio-temporal correlation in the skin-stimulation pattern. To elucidate this mechanism, we presented, on braille displays, tactile random-dot kinematograms, similar to those widely used in visual motion research, which enables us to independently manipulate feature trackability and various parameters of local motion. We found that a human observer is able to detect the direction of difficult-to-track tactile motions presented to the fingers and palms. In addition, the direction-discrimination performance was better when the stimuli were presented along the fingers than when presented across the fingers. These results indicate that low-level motion sensing, in addition to high-level tracking, contribute to tactile motion perception.

Describing the Sensation of the 'Velvet Hand Illusion' in Terms of Common Materials.

When sandwiching two moving parallel metallic wires between both hands, one often experiences an unexpected tactile sensation known as the "velvet hand illusion" (VHI). Researchers have revealed the optimal conditions for inducing VHI, while the subjective nature of VHI remains obscure. In this article, we conducted a psychophysical experiment to investigate the quality and magnitude of the illusory sensation felt during VHI. Participants were asked to evaluate the tactile sensation of moving wires by giving tactile adjective and intensity ratings of the illusory sensation. In the same experiment, for the sake of comparison, participants also rated the sensation for various common materials one may encounter in daily life. We found that, as the intensity of the illusory sensation increased, the tactile sensation became softer, wetter, warmer, and more favorable. We also found that, when a strong illusion was reported, the sensation was similar to those for leather and fabrics rather than metallic wire, which suggests that the illusion indeed changes the perceived material category. These findings provide a better characterization of VHI as well as a better understanding of tactile texture perception.

An Android for Emotional Interaction: Spatiotemporal Validation of Its Facial Expressions.

Android robots capable of emotional interactions with humans have considerable potential for application to research. While several studies developed androids that can exhibit human-like emotional facial expressions, few have empirically validated androids' facial expressions. To investigate this issue, we developed an android head called Nikola based on human psychology and conducted three studies to test the validity of its facial expressions. In Study 1, Nikola produced single facial actions, which were evaluated in accordance with the Facial Action Coding System. The results showed that 17 action units were appropriately produced. In Study 2, Nikola produced the prototypical facial expressions for six basic emotions (anger, disgust, fear, happiness, sadness, and surprise), and naïve participants labeled photographs of the expressions. The recognition accuracy of all emotions was higher than chance level. In Study 3, Nikola produced dynamic facial expressions for six basic emotions at four different speeds, and naïve participants evaluated the naturalness of the speed of each expression. The effect of speed differed across emotions, as in previous studies of human expressions. These data validate the spatial and temporal patterns of Nikola's emotional facial expressions, and suggest that it may be useful for future psychological studies and real-life applications.

Visual perception of liquids: Insights from deep neural networks.

Visually inferring material properties is crucial for many tasks, yet poses significant computational challenges for biological vision. Liquids and gels are particularly challenging due to their extreme variability and complex behaviour. We reasoned that measuring and modelling viscosity perception is a useful case study for identifying general principles of complex visual inferences. In recent years, artificial Deep Neural Networks (DNNs) have yielded breakthroughs in challenging real-world vision tasks. However, to model human vision, the emphasis lies not on best possible performance, but on mimicking the specific pattern of successes and errors humans make. We trained a DNN to estimate the viscosity of liquids using 100.000 simulations depicting liquids with sixteen different viscosities interacting in ten different scenes (stirring, pouring, splashing, etc). We find that a shallow feedforward network trained for only 30 epochs predicts mean observer performance better than most individual observers. This is the first successful image-computable model of human viscosity perception. Further training improved accuracy, but predicted human perception less well. We analysed the network's features using representational similarity analysis (RSA) and a range of image descriptors (e.g. optic flow, colour saturation, GIST). This revealed clusters of units sensitive to specific classes of feature. We also find a distinct population of units that are poorly explained by hand-engineered features, but which are particularly important both for physical viscosity estimation, and for the specific pattern of human responses. The final layers represent many distinct stimulus characteristics-not just viscosity, which the network was trained on. Retraining the fully-connected layer with a reduced number of units achieves practically identical performance, but results in representations focused on viscosity, suggesting that network capacity is a crucial parameter determining whether artificial or biological neural networks use distributed vs. localized representations.

Direction of Apparent Motion During Smooth Pursuit Is Determined Using a Mixture of Retinal and Objective Proximities.

Many studies have investigated various effects of smooth pursuit on visual motion processing, especially the effects related to the additional retinal shifts produced by eye movement. In this article, we show that the perception of apparent motion during smooth pursuit is determined by the interelement proximity in retinal coordinates and also by the proximity in objective world coordinates. In Experiment 1, we investigated the perceived direction of the two-frame apparent motion of a square-wave grating with various displacement sizes under fixation and pursuit viewing conditions. The retinal and objective displacements between the two frames agreed with each other under the fixation condition. However, the displacements differed by 180 degrees in terms of phase shift, under the pursuit condition. The proportions of the reported motion direction between the two viewing conditions did not coincide when they were plotted as a function of either the retinal displacement or of the objective displacement; however, they did coincide when plotted as a function of a mixture of the two. The result from Experiment 2 showed that the perceived jump size of the apparent motion was also dependent on both retinal and objective displacements. Our findings suggest that the detection of the apparent motion during smooth pursuit considers the retinal proximity and also the objective proximity. This mechanism may assist with the selection of a motion path that is more likely to occur in the real world and, therefore, be useful for ensuring perceptual stability during smooth pursuit.

Perceptually Based Adaptive Motion Retargeting to Animate Real Objects by Light Projection.

A recently developed light projection technique can add dynamic impressions to static real objects without changing their original visual attributes such as surface colors and textures. It produces illusory motion impressions in the projection target by projecting gray-scale motion-inducer patterns that selectively drive the motion detectors in the human visual system. Since a compelling illusory motion can be produced by an inducer pattern weaker than necessary to perfectly reproduce the shift of the original pattern on an object's surface, the technique works well under bright environmental light conditions. However, determining the best deformation sizes is often difficult: When users try to add a large deformation, the deviation in the projected patterns from the original surface pattern on the target object becomes apparent. Therefore, to obtain satisfactory results, they have to spend much time and effort to manually adjust the shift sizes. Here, to overcome this limitation, we propose an optimization framework that adaptively retargets the displacement vectors based on a perceptual model. The perceptual model predicts the subjective inconsistency between a projected pattern and an original one by simulating responses in the human visual system. The displacement vectors are adaptively optimized so that the projection effect is maximized within the tolerable range predicted by the model. We extensively evaluated the perceptual model and optimization method through a psychophysical experiment as well as user studies.

Deformation-induced transparency resolves color scission.

Dynamic image deformation produces the perception of a transparent material that appears to deform the background image by light refraction. Since past studies on this phenomenon have mainly used subjective judgment about the presence of a transparent layer, it remains unsolved whether this is a real perceptual transparency effect in the sense that it forms surface representations, as do conventional transparency effects. Visual computation for color and luminance transparency, induced mainly by surface-contour information, can be decomposed into two components: surface formation to determine foreground and background layers, and scission to assign color and luminance to each layer. Here we show that deformation-induced perceptual transparency aids surface formation by color transparency and consequently resolves color scission. We asked observers to report the color of the front layer in a spatial region with a neutral physical color. The layer color could be seen as either reddish or greenish depending on the spatial context producing the color transparency, which was, however, ambiguous about the order of layers. We found that adding to the display a deformation-induced transparency that could specify the front layer significantly biased color scission in the predicted way if and only if the deformation-induced transparency was spatially coincident with the interpretation of color transparency. The results indicate that deformation-induced transparency is indeed a novel type of perceptual transparency that plays a role in surface formation in cooperation with color transparency.

Motion Perception: From Detection to Interpretation.

Visual motion processing can be conceptually divided into two levels. In the lower level, local motion signals are detected by spatiotemporal-frequency-selective sensors and then integrated into a motion vector flow. Although the model based on V1-MT physiology provides a good computational framework for this level of processing, it needs to be updated to fully explain psychophysical findings about motion perception, including complex motion signal interactions in the spatiotemporal-frequency and space domains. In the higher level, the velocity map is interpreted. Although there are many motion interpretation processes, we highlight the recent progress in research on the perception of material (e.g., specular reflection, liquid viscosity) and on animacy perception. We then consider possible linking mechanisms of the two levels and propose intrinsic flow decomposition as the key problem. To provide insights into computational mechanisms of motion perception, in addition to psychophysics and neurosciences, we review machine vision studies seeking to solve similar problems.

Estimating Tactile Perception by Observing Explorative Hand Motion of Others.

When we acquire tactile information about an object's surface, we actively move our hands. Past studies have shown a correlation between participants' (i.e., touchers') hand motion in tactile exploration and the evaluated tactile attributes of the object, which suggests that tactile perception can be estimated from statistical analysis of touchers' hand motion. Though it has been reported that the statistical analysis of hand motion can indeed estimate tactile perception, whether humans can estimate tactile perception by observing the hand motion of others remains unclear. To investigate this, we conducted experiments wherein observers watched point-light moving hands of touchers in tactile exploration and evaluated the material being touched. Our results show that, although observers' estimation of touchers' perception was not accurate, observers extracted information from touchers' hand motion for estimation, and the correlations within observers' estimation were high. These results suggest that human observers can estimate tactile perception through visual observation of the hand motion of others by adopting common strategies about the relationships between touchers' hand motion and tactile perception.

Material and shape perception based on two types of intensity gradient information.

Visual estimation of the material and shape of an object from a single image includes a hard ill-posed computational problem. However, in our daily life we feel we can estimate both reasonably well. The neural computation underlying this ability remains poorly understood. Here we propose that the human visual system uses different aspects of object images to separately estimate the contributions of the material and shape. Specifically, material perception relies mainly on the intensity gradient magnitude information, while shape perception relies mainly on the intensity gradient order information. A clue to this hypothesis was provided by the observation that luminance-histogram manipulation, which changes luminance gradient magnitudes but not the luminance-order map, effectively alters the material appearance but not the shape of an object. In agreement with this observation, we found that the simulated physical material changes do not significantly affect the intensity order information. A series of psychophysical experiments further indicate that human surface shape perception is robust against intensity manipulations provided they do not disturb the intensity order information. In addition, we show that the two types of gradient information can be utilized for the discrimination of albedo changes from highlights. These findings suggest that the visual system relies on these diagnostic image features to estimate physical properties in a distal world.

Human tactile detection of within- and inter-finger spatiotemporal phase shifts of low-frequency vibrations.

When we touch an object, the skin copies its surface shape/texture, and this deformation pattern shifts according to the objects movement. This shift pattern directly encodes spatio-temporal "motion" information of the event, and has been detected in other modalities (e.g., inter-aural time differences for audition and first-order motion for vision). Since previous studies suggested that mechanoreceptor-afferent channels with small receptive field and slow temporal characteristics contribute to tactile motion perception, we tried to tap the spatio-temporal processor using low-frequency sine-waves as primitive probes in our previous study. However, we found that asynchrony of sine-wave pair presented on adjacent fingers was difficult to detect. Here, to take advantage of the small receptive field, we investigated within-finger motion and found above threshold performance when observers touched localized sine-wave stimuli with one finger. Though observers could not perceptually discriminate rightward from leftward motion, the adaptation occurred in a direction-sensitive way: the motion/asynchronous detection was impaired by adapting to asynchronous stimuli moving in the same direction. These findings are consistent with a possibility that human can directly encode short-range spatio-temporal patterns of skin deformation by using phase-shifted low-frequency components, in addition to detecting short- and long-range motion using energy shifts of high-frequency components.

Reduction of Flicker in Four-Stroke Motion of Color Images.

When two sequential video frames extracted from a single video clip are followed by the negative of the two frames, a viewer often experiences a visual illusion whereby a scene in the frames continuously moves in a single direction (four-stroke apparent motion). To create a four-stroke apparent motion display, the image intensities of the whole of the second pair of images are reversed. However, this intensity reversal creates a strong impression of flicker that can be undesirable for comfortable viewing. This study reports that four-stroke apparent motion can be induced by only reversing the luminance intensities in those spatial areas which contain motion signals in high-pass filtered images. This use of only a partial reversal of image intensities greatly reduces the apparent flicker in the display while retaining motion perception.

Contour junctions defined by dynamic image deformations enhance perceptual transparency.

The majority of work on the perception of transparency has focused on static images with luminance-defined contour junctions, but recent work has shown that dynamic image sequences with dynamic image deformations also provide information about transparency. The present study demonstrates that when part of a static image is dynamically deformed, contour junctions at which deforming and nondeforming contours are connected facilitate the deformation-based perception of a transparent layer. We found that the impression of a transparent layer was stronger when a dynamically deforming area was adjacent to static nondeforming areas than when presented alone. When contour junctions were not formed at the dynamic-static boundaries, however, the impression of a transparent layer was not facilitated by the presence of static surrounding areas. The effect of the deformation-defined junctions was attenuated when the spatial pattern of luminance contrast at the junctions was inconsistent with the perceived transparency related to luminance contrast, while the effect did not change when the spatial luminance pattern was consistent with it. In addition, the results showed that contour completions across the junctions were required for the perception of a transparent layer. These results indicate that deformation-defined junctions that involve contour completion between deforming and nondeforming regions enhance the perception of a transparent layer, and that the deformation-based perceptual transparency can be promoted by the simultaneous presence of appropriately configured luminance and contrast-other features that can also by themselves produce the sensation of perceiving transparency.

Grouping by feature of cross-modal flankers in temporal ventriloquism.

Signals in one sensory modality can influence perception of another, for example the bias of visual timing by audition: temporal ventriloquism. Strong accounts of temporal ventriloquism hold that the sensory representation of visual signal timing changes to that of the nearby sound. Alternatively, underlying sensory representations do not change. Rather, perceptual grouping processes based on spatial, temporal, and featural information produce best-estimates of global event properties. In support of this interpretation, when feature-based perceptual grouping conflicts with temporal information-based in scenarios that reveal temporal ventriloquism, the effect is abolished. However, previous demonstrations of this disruption used long-range visual apparent-motion stimuli. We investigated whether similar manipulations of feature grouping could also disrupt the classical temporal ventriloquism demonstration, which occurs over a short temporal range. We estimated the precision of participants' reports of which of two visual bars occurred first. The bars were accompanied by different cross-modal signals that onset synchronously or asynchronously with each bar. Participants' performance improved with asynchronous presentation relative to synchronous - temporal ventriloquism - however, unlike the long-range apparent motion paradigm, this was unaffected by different combinations of cross-modal feature, suggesting that featural similarity of cross-modal signals may not modulate cross-modal temporal influences in short time scales.